REPLACEMENT SENSOR MODEL FOR OPTIMAL IMAGE 
EXPLOITATION 



Related Applications 

[0001] This application claims priority to US Provisional Application No. 60/305,440, filed 
July 13, 2001, the entire content of which is hereby incorporated by reference 

Background of the Invention 

Field of the Invention 

[0002] The present invention relates to sensor models for describing a ground-to-image 
relationship for image sensors. More specifically, the present invention relates to a 
replacement sensor model that is generic to a plurality of image sensors and that reliably 
provides a full range of image exploitation. 

Description of the Related Art 

[0003] Image sensors are widely-known tools for obtaining images of various objects. More 
particularly, image sensors such as airborne or space-borne image sensors are known to be 
useful in obtaining images of underlying objects as the sensors pass by or over the objects. 
Various characteristics of these objects that are demonstrated by the resultant images can then 
be observed, including a temperature of the object, its position (latitude, longitude and/or 
elevation), its appearance, its color, its relation to another object, etc. 
[0004] Since there exist these various characteristics of the imaged objects that can be 
observed, the term "image exploitation" is generically used to refer to a use to which an 
image is put. A subset of image exploitation is referred to as "geopositioning," where, for 
example, an image is exploited by determining a latitude and longitude of the imaged object 
based on the image. Another example is determining latitude and longitude along with 
elevation of the image object based on multiple images. Geopositioning is considered 
"optimal" when it is performed in a manner such that its solution errors are minimized. This, 
in turn, requires statistical knowledge of the errors affecting the solution, such as the error in 
the knowledge of the sensor's position when the image was taken. 

[0005]Image sensors such as those just discussed have various internal characteristics such as 
focal length, as well as various external characteristics such as location and attitude of the 
sensor at a given point in time. These characteristics, which vary individually by sensor, are 
the typical sources of error just referred to. These sources of error must be accounted for in 
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order to obtain reliable and accurate image information, and/or to be aware of a possible 
extent of any resulting errors. Such characteristics and their specific values are 
conventionally referred to as "support data," or "sensor support data." The support data can 
be obtained by, for example, receipt of navigational data transmitted from the image sensor's 
platform to a ground station. For example, data may be obtained from a Global Positioning 
System (GPS) receiver on board a satellite. 

[0006lAn imaging methodology that reflects the above-discussed concepts is known as a 
"sensor model," or, more specifically, a "rigorous (or physical) sensor model." Specifically, 
a rigorous sensor model for an image sensor such as the one discussed above relates ground 
coordinates (i.e., three-dimensional data pertaining to the imaged object) to image 
coordinates (i.e., two-dimensional pixel data for the image of the object), utilizing the sensor 
support data that accompany each image and that specify sensor characteristics at a time the 
image was obtained, as discussed above. 

[0007] As demonstrated in Fig. 1, a sensor model 110 consists of a ground-to-image 
transformation; i.e., a mathematical function with a three-dimensional ground coordinate set 
120 as input and a two-dimensional image (pixel) location as output 130. The ground-to- 
image transformation of sensor model 1 10 is parameterized by the support data and as such 
is descriptive of the sensor's characteristics. The transformation of ground coordinates into 
image coordinates effected by sensor model 1 10 is thus described by Equation (1): 

u = F(X,S)andv = G(X,S), (1) 

where u and v are the image pixel coordinates, X is a three dimensional vector describing the 
ground data as discussed above, and S is the sensor support data state vector (the error 
covariance shown in Fig. 1 is discussed in detail below). 

[0008] The sensor model 1 10 is routinely used in image exploitation. As an example of this 
type of use of the sensor model 110, the geopositioning process referred to above will be 
discussed in more detail. 

[0009] Proper geopositioning conventionally exists of two types of solutions: ground point 
(target) extraction and triangulation. The first of these two is obtained by the Extraction 
Process, where the location of ground point(s) is extracted from the two dimensional image 
coordinates of the object identified and measured in the image(s). For example, an operator 
of an image sensor within a satellite may obtain a plurality of images of relatively large areas 
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on the earth's surface, such as a city or a portion of a city. Such an area of the entire image is 
typically known as the "image footprint." Corresponding image data may then be stored by 
the operator, along with the support data of the image sensor applicable at the time the image 
was taken. In this way, a location of a particular target object such as a particular building 
within the city may be "extracted" from the image data. 

[0010] In order to perform this process, for example using one image, an inverse 
transformation of the sensor model 110 and associated equation (1) is used; i.e., pixel data 
(u,v) and the vertical component of X, known a priori, are used as inputs to obtain the two 
horizontal components of X as outputs. The inverse transformation is typically a numerical, 
iterative inverse of the ground-to-image function. The support data state vector S is treated 
as a vector of fixed parameters. Examples of the Extraction Process are discussed in more 
detail below with respect to Figs. 2-5. 

10011} The inverse transformation is a straight-forward solution technique applicable to one 
image. However, in order to also provide estimates of the one image solution accuracy, as 
well as utilize more than one image, a more sophisticated approach is required. This 
approach is characterized by the "M-image Extraction Algorithm." This algorithm is 
discussed in detail below; generally speaking, however, the algorithm utilizes various image 
data corresponding to a plurality of images of the same target object, at least some of which 
offer different perspectives of the object. 

[00121 Multiple perspectives are required to extract three dimensional ground positions from 
two dimensional images. Also, each of the images are weighted (criteria for performing the 
weighting is also discussed below), and the ground point locations are solved for iteratively 
in an optimal fashion (i.e., an estimate of X is used to begin iteration of the algorithm leading 
to progressively more accurate estimates of X). Solution accuracy estimates are also 
provided. 

[0013J Fig. 2 illustrates the role of the sensor model's ground-to-image transformation in the 
extraction of a ground point 210 using a stereo pair of images 220 and 230. In Fig. 2, images 
220 and 230 may either be obtained by two separate image sensors (perhaps at the same 
point in time), or by a single image sensor at different points in time. In Fig. 2, the ground- 
to-image transformation (or its inverse) specifies the image rays. The image rays intersect 
the image coordinates measured in each image. 

[0014] Fig. 2 assumes perfect sensor support data. However, as discussed above, there will 
virtually always be errors in the sensor support data, as illustrated by incorrect sensor 
position/orientation 310 in Figure 3, and any such errors will propagate to errors in extracted 



ground points. Thus, the support data errors should be accounted for during extraction in 
some manner in order to achieve optimal image exploitation (i.e., here, optimal 
geopositioning). 

[0015] One way of accounting for support data errors is to determine a statistical "outer 
bound" or possible extent of the support data errors as they propagate to errors in the 
extracted ground points. For example, support data may indicate that a satellite (and its 
image sensor) was in a certain position at the time a given image was taken. An error in this 
data (i.e., the satellite was not actually at that exact position) may result is an extracted 
ground point being calculated as being, for example, up to ten meters off from its actual 
location. Such knowledge of this extent to which the support data (and thus the determined 
position of the extracted ground point) is inaccurate is referred to as "error propagation." 
Error propagation can also refer to related concepts, such as, for example, a determination 
that a relative position between two particular imaged objects is accurate to within a 
particular distance. 

[0016] Additionally, if a given image sensor obtains multiple images of a single object, or 
multiple image sensors each obtain a single image of an object(s), it would be useful to know 
which ones of the images are associated with more accurate support data. This knowledge is 
the primary criteria mentioned above that allows for the weighting of the multiple images in 
the M-image Extraction Algorithm, so that the more accurate images are given relatively 
more importance in arriving at a composite solution. 

[0017] In short, it is useful to know an extent to which the support data is accurately known; 
i.e., an extent of errors contained in the support data. This knowledge is quantified by an 
"error covariance" matrix. Specifically, the quantification of the accuracy of sensor support 
data is represented by an a priori (initial) error covariance matrix of the support data (see Fig. 
1). In its most general form, the a priori error covariance of the sensor support data errors is 
an (ran x mn) matrix, where m equals the number of images and n is the number of sensor 
support data error components per image. 

[0018] The accuracy of the extracted ground point's location is quantified by its a posteriori 
(after solution) error covariance matrix. The a posteriori error covariance matrix is primarily 
a function of the number of images, their geometry, and the a priori error covariance matrix 
of the sensor support data. Along with the ground point solution, the a posteriori error 
covariance is also output from the M-image Extraction Algorithm. 

[0019] Equation (2) formally defines the error covariances (all error processes are assumed 
unbiased, i.e., zero mean, and E{} corresponds to expected value): 
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C s = E{sSsS T } , the sensor state vector a priori error covariance 
P = E{sXeX t } , the extracted ground point a posteriori error covariance, 
where sS and sK are the error vectors (multi - variate random variables) 
associated with S and X 



|0020] Figure 4 illustrates the extraction of a ground point using multiple images from 
multiple image sensor positions 410a-410n, as might be performed by the M-image 
Extraction Algorithm. As in Figs. 2 and 3, the ground-to-image transformation of the 
rigorous sensor model can be used to specify the image rays. Sensor support data errors 
prevent the multiple rays from intersecting at a common point, but the error covariance 
associated with the rigorous sensor model can quantify an extent of the errors, and thereby 
determine which rays correspond to more accurate support data. In this way, these rays can 
be given more weight in executing the algorithm. Generally, the more images utilized, the 
more accurate the ground point extraction (solution) will be. 

[0021] Figs. 2-4 have depicted only two of the three ground (object space) dimensions for 
simplicity. Images 510 and 520 in Fig. 5 further illustrate exemplary effects of support data 
errors, but in all three dimensions. Again, the image rays do not intersect due to the errors, 
but the error covariance will quantify an extent of the errors and thereby allow for error 
propagation and weighting of the images to obtain an optimal ground point extraction. In 
this particular example, both images are given equal weight; hence, the solution is at the 
midpoint of the minimum separation vector. 

[0022] The above discussion has focused on the Extraction Process in performing optimal 
geopositioning. However, as mentioned above, geopositioning may also include the 
Triangulation Process. 

[0023] In the Extraction Process, a location of a ground point(s) is extracted from image data, 
and an extent of support data errors, quantified by an error covariance matrix, permits error 
propagation and proper weighting of a plurality of images in obtaining an optimal ground 
point solution. The Triangulation Process also solves for the ground points, but additionally 
actually solves for the support data errors of all the images involved, and adjusts or reduces 
them so that a corresponding error(s) in the ground point solution or in subsequent 
extractions is actually reduced. 

[0024] Solving for the support data errors according to the Triangulation Process typically 
requires additional information to be input into the process. For example, a location (ground 



point) of a control point within the image may be previously known to a great degree of 
accuracy, and can thus be used as a reference to actually solve for (and correct) errors in the 
sensor support data. Triangulation, as is known, typically involves a weighted least squares 
adjustment, using the a priori error covariance of the support data and the control points, 
and/or other previously-known information, to adjust the support data. 

[0025] In summary, a complete sensor model consists of a ground-to-image transformation, 
the sensor support data, and the a priori error covariance of the sensor support data. The 
error covariance is relative to an identified set of sensor parameters. The relevant sensor 
support data errors are the dominant errors, such as sensor position error and attitude error. 
The support data a priori error covariance matrix quantifies the expected magnitude of the 
errors, the correlation between different error components from the same image, and, when 
applicable, the cross correlation between errors from different images (i.e., the extent to 
which errors in different images are related to one another). Error covariance can be used for 
extraction, including error propagation and weighting of a plurality of images. If additional 
information is known with respect to, for example, control points within the image footprint, 
error covariance can also allow solving for the sensor support data errors and adjusting them 
so as to obtain a more accurate determination of a target ground point(s). 
[0026] Using the above techniques, rigorous sensor models are conventionally recognized as 
providing a ground-to-image transformation in support of optimal image exploitation, 
including geopositioning. However, since such rigorous sensor models are specific to 
particular image sensors, users of the image sensors must have access to the specific models. 
If a user is utilizing many different image sensors, the costs associated with obtaining, 
maintaining and utilizing all of the associated rigorous sensor models becomes burdensome. 
Moreover, many operators of image sensors do not wish to essentially share proprietary 
information relating to their equipment by distributing rigorous sensor models to users. 
[0027] Therefore, a more general sensor model, termed the abstract sensor model, has been 
developed. It is also a ground-to-image function, and is conventionally expressed as a 
polynomial as shown in equation (3), although a ratio of two polynomials, commonly known 
as a rational function, is sometimes used: 



u = aO + alx + a2y + a3z + a4xy + ... 
v = bO + blx + b2y + b3z + b4xy + .. 



(3) 



In equation (3), x,y,z are the components of the ground position vector X and aO,al,..,bO,bl,.. 
are the polynomial coefficients. Typically, X is relative to a ground coordinate system that is 
centered at the middle of the image footprint and scaled such that the coordinate values x, y, 
and z range from -1 .0 to 1 .0. 

[0028] The ground-to-image function corresponds to the polynomial and the "support data" 
are the values of the polynomial coefficients. Such a conventional abstract sensor model has 
various advantages. For example, the user need not know nor obtain the specific rigorous 
sensor models. Thus, one such model can be used for many different sensors, which also 
affords a common interface for the distribution and use of imagery and its support data (since 
a given user only needs software corresponding to the polynomial, and can then simply "plug 
in" the values of the coefficients and image data). 

[0029] Moreover, these advantages result in lower costs for user development and 
maintenance. The use of an abstract sensor model polynomial defining a conventional 
abstract sensor model can also result in higher user throughput, due to its usually faster 
evaluation as compared to the original sensor model. Finally, the use of a polynomial can 
lead to wider availability of a specific sensor's imagery, since the polynomial coefficients 
(unlike the actual support data) do not have any obvious, literal physical meaning and 
therefore do not provide detail into proprietary sensor characteristics (as the original sensor 
model does). 

[0030] In generating coefficients for the polynomial, the rigorous sensor model is used to 
generate a grid of image pixel-ground point correspondences across a representative area of 
the image footprint, where the ground points correspond to multiple elevation planes. That 
is, a number of latitude, longitude and elevation points over a representative area of the 
footprint are selected and inserted into the ground-to-image function to generate 
corresponding image points. (Alternatively, a number of image points are selected and 
inserted into the corresponding image-to-ground function, along with an a priori elevation, to 
generate corresponding ground points.) The coefficients are then fit to the grid of resulting 
correspondences to obtain the coefficients' values. The original sensor model can use either 
original sensor support data or triangulated sensor support data. Any support data errors that 
still exist within the sensor model when generating the polynomial coefficients will be 
reflected within the coefficients. However, like the coefficients themselves, such errors will 
not have any apparent physical meaning and so can not conventionally be accurately 
quantified by an associated error covariance. 

[0031] A straightforward method for performing the fit to determine the coefficients is the 
well-known least squares fitting process. The size of the "fit error," i.e., the difference in 



image coordinates between the polynomial evaluation and the original sensor model 
evaluation at a common ground point(s), depends on the form (degree) of the polynomial and 
the details of the fitting process. Such details of generating a conventional abstract sensor 
model polynomial as just described are conventionally well-known. 

[0032] Equation (4) illustrates a typical polynomial abstract sensor model. It is order 2 in x, 
order 2 in y, and order 1 in z. Typical polynomial fit error is sub-pixel, with 0.01 pixel root- 
mean-square error not uncommon. 

u = EiSvVz k and v = £i£b Ijk xyz k (4) 

i=0 j=0 k=0 i=° j=° k=0 

[0033] Conventional abstract sensor models can match the ground-to-image relationship of 
an original sensor model with a great deal of accuracy for many image sensors. That is, they 
are generally capable of providing very similar image outputs when given the same ground 
point inputs (or vice-versa). Thus, such abstract sensor models can be used to perform a 
rudimentary extraction of a ground point target object from an image(s) as discussed above. 
[0034] However, abstract sensor models have no associated error covariance that is 
equivalent in any meaningful way with the sensor support data error covariance of the 
rigorous sensor model. Without such an error covariance, conventional abstract sensor 
models can not perform useful error propagation, determine optimal weights to assign 
multiple images in determining a composite solution, or solve/correct for "support data" 
errors (i.e., support data errors reflected in the polynomial coefficients) in a triangulation 
process. 

[00351 What is needed is an abstract model that provides an accurate but flexible ground-to- 
image relationship that is easy to use and applicable to many sensors, such as that provided 
by conventional abstract sensor models, together with a meaningful error covariance relative 
to a set of adjustable parameters that allows for optimal image exploitation, such as that 
conventionally provided by rigorous sensor models. 

Summary of the Invention 
[0036] The present invention relates to a method for generating a replacement sensor model 
(RSM) data file for use in processing image data produced by an image sensor. The method 
may include: (1) generating parameters (e.g., polynomial coefficients) of a ground-to-image 
function based on a sensor model of the image sensor, the sensor model including support 
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data describing physical characteristics of the image sensor, (2) defining adjustable 
parameters of the ground-to-image function to reflect errors in the support data, and (3) 
generating an error covariance relative to these adjustable parameters and containing 
information virtually equivalent to information contained in a support data error covariance. 
According to this method, the data file includes the identification and values of the ground- 
to-image function parameters, adjustable parameters, and adjustable parameters error 
covariance. 

[0037] The present invention additionally relates to a system for permitting exploitation of 
imaging data by a plurality of users. The system may include an image sensor, a transmitter 
operable to transmit images obtained by the image sensor to the plurality of users and a 
replacement sensor model (RSM) data file generator for computing parameters of a 
replacement sensor model, including parameters of a ground-to-image function that is 
adjustable and an RSM error covariance associated with the adjustable function so as to 
reflect errors in support data describing the image sensor. One data file is generated per 
image and also transmitted to the plurality of users. 

[0038] Additionally, the present invention relates to a method for exploiting imaging data. 
The method may include inputting, into a processor, replacement sensor model data files, 
each containing an abstract ground-to-image function for a generic image sensor, as well as 
an image obtained by an image sensor. The method may also include using the processor to 
calculate a position of a chosen ground point(s) within the images using the data files. In this 
method of the invention, a data file includes parameters of the ground-to-image function, a 
set of adjustable parameters (adjustment vector) operable to adjust the function to correct for 
errors in the data, and an error covariance corresponding to the vector of adjustable 
parameters. 

[0039] Finally, the present invention also relates to a replacement sensor model (RSM). The 
replacement sensor model includes a ground-to-image function, the function being generated 
from data describing physical characteristics of the image sensor, an adjustment vector for 
adjusting the function so as to reflect errors in the data, and an error covariance associated 
with the adjustment vector that contains virtually identical information as an error covariance 
of a rigorous sensor model of the image sensor. 

[0040] The features and advantages of the invention will become apparent from the following 
drawings and description. 



Brief Description of the Drawings 
[0041] The present invention is described with reference to the accompanying drawings. In 
the drawings, like reference numbers indicate identical or functionally similar elements. 
Additionally, the left-most digit(s) of a reference number identifies the drawing in which the 
reference number first appears. 

[0042] Fig. 1 illustrates a rigorous sensor model of the prior art. 

[0043] Fig. 2 illustrates the role of the ground-to-image transformation of the sensor model of 
Fig. 1 in the extraction of a three dimensional ground point using a stereo pair of images. 
[0044] Fig. 3 illustrates an effect of erroneous image sensor support data on a ground point 
stereo solution of Fig. 2. 

[0045] Figure 4 illustrates the extraction of a ground point using multiple images. 

[0046] Figure 5 further illustrates an example of the effects of support data errors in three 

dimensions. 

[0047] Fig. 6 demonstrates an exemplary implementation of one embodiment of the present 
invention. 

[0048] Fig. 7 is a flowchart describing the implementation of Fig. 6. 

[0049] Fig. 8A demonstrates a ground space adjustment to the ground-to-image polynomial 
according to one embodiment of the present invention. 

[0050] Fig. 8B demonstrates an image space adjustment to the ground-to-image polynomial 
according to one embodiment of the present invention. 

Detailed Description 

[0051] While the present invention is described below with respect to various exemplary 
embodiments, the present invention is not limited to only those embodiments that are 
disclosed. Other embodiments can be implemented by those skilled in the art without 
departing from the spirit and scope of the present invention. 

[0052] In one exemplary embodiment, the present invention includes an abstract sensor 
model that is capable of all the functionality of a rigorous sensor model and provides greatly 
improved image exploitation over conventional abstract sensor models, and that includes 
adjustability and an error covariance containing information equivalent to the information 
contained in a sensor support data error covariance. 

[0053] Such an abstract sensor model is termed a replacement sensor model (RSM). A 
replacement sensor model according to one embodiment of the present invention includes a 
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.d-to-image mathematical transformation function for converting three-dimensional 
und point data into two-dimensional image (pixel) data. The function, by itself, is 
.pplicable to virtually any sensor. The function could include, for example, a polynomial, a 
ratio of polynomials or an interpolated grid of ground point-image point correspondences. 
Parameters of the function (e.g., coefficients of the polynomial) are determined by fitting the 
parameters to the support data of a particular image sensor; i.e., by fitting the parameters to a 
ground point-image point correspondence grid corresponding to an image footprint and 
mathematically generated using the rigorous sensor model. 

[0054] Types of adjustable parameters are selected for the mathematical function that relate 
to types/sources of support data error and that permit the calculation of an RSM error 
covariance containing information equivalent to information contained in the sensor support 
data error covariance of the rigorous sensor model. The parameters may be thought of as 
forming an adjustment vector A, and together allow for the generation of the RSM error 
covariance. 

[0055J Since the actual support data errors are unknown, the actual values of the adjustable 
parameters (i.e., the adjustment vector) will be zero during an initial iteration of, for example, 
a ground point extraction algorithm. Nevertheless, the existence of the error covariance 
associated with the replacement sensor model and its adjustable parameters, once generated, 
allows a user of the RSM to practice error propagation and proper weighting of a plurality of 
images in obtaining an optimal solution. Moreover, a process similar to triangulation allows 
a user to solve for and assign non-zero values to the adjustment parameters (vector A), 
whereby solution accuracy can be improved. Here, such users can then disseminate the 
improved RSM, including the new A having non-zero values, for use by subsequent users. 
[0056] Thus, once the adjustable parameters are selected, the corresponding partial 
derivatives are calculated therewith, and the RSM covariance is in turn calculated from the 
corresponding partial derivatives and the rigorous sensor model error covariance. Thereafter, 
an actual implementation of the invention becomes straightforward, requiring only the 
substitution of the RSM error covariance, the adjustable parameter (definitions), and the 
parameterized polynomial ground-to-image function for the corresponding entities in the 
rigorous sensor model. The actual calculations then proceed in virtually the same manner as 
in the prior art rigorous sensor models, and provide potentially identical results, but with all 
the aforementioned advantages of abstract sensor models. 

[0057] In short, this embodiment of the present invention seeks to translate all of the 
information contained within a support data error covariance of a conventional rigorous 
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sensor model into an error covariance associated with an abstract sensor model. Together, 
the abstract sensor model and its associated error covariance (and an adjustment vector 
instrumental in the translation of the error covariance information) make up a replacement 
sensor model (RSM) that provides image exploitation, including geopositioning, that is 
virtually identical to the rigorous sensor model of the prior art but retains all of the 
advantages of a conventional abstract sensor model. 

[0058] The following is a brief review of concepts and terms that will be helpful in 
understanding the detailed description of various exemplary embodiments discussed below. 
Specifically, it should be understood from the above Background section that a support data 
error covariance of a conventional rigorous sensor model (hereafter referred to as C s ) 
quantifies a possible extent of errors present in the support data. The support data and 
included errors have physical meaning in that they demonstrate actual characteristics of an 
operator's image sensor. 

[0059] When an abstract sensor model is generated, the support data (and all the 
corresponding errors) are translated into the parameters of that model's transformation 
function (e.g., coefficients of a polynomial). At this point, the parameters still contain the 
same information (including errors) as the support data, but they no longer have any physical 
meaning. This loss of physical meaning, as discussed, is partially by design, and prevents the 
distribution of proprietary image sensor characteristics. Nevertheless, the translation of the 
physical support data into the abstract function parameters essentially camouflages the error 
information along with the actual support data, and, until now, has thereby prevented the 
generation of a useful error covariance associated with an abstract sensor model. 
[0060] In an exemplary embodiment of the present invention, therefore, the characteristics of 
the support data and its errors of the rigorous sensor model are projected into "image space." 
In other words, a physical error on the "ground" or "sensor" side of the model is taken, such 
as might be represented by a position error of an image sensor corresponding to a ten meter 
error in ground location. This error is projected, or translated, onto the "image" side of the 
model, so that the ten meter ground error effect might be found to correspond to, for example, 
a twenty pixel image error. 

[0061] This embodiment then projects information from the "ground" or "sensor" side of an 
abstract sensor model into image space. The difference between the two projections is then 
minimized; that is, adjustments to the transformation function of the RSM can be identified 
that correlate to a twenty pixel error, thus reclaiming the error information that was lost in the 
translation of the rigorous sensor model into the abstract sensor model parameters as 
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discussed above. An error covariance is then generated based on the statistical characteristics 
of the determined adjustments. In this way, an RSM error covariance containing virtually all 
of the information in the rigorous sensor model error covariance can be generated. 
[0062J Once the RSM error covariance is generated, it can be used within the RSM in 
virtually the same manner that the prior art rigorous sensor model utilized its associated error 
covariance. That is, once generated, parameters of the present invention can often be used 
simply by substituting them for their prior art counterparts in conventional algorithms and 
equations. 

[0063] The following describes in more detail the generation and operation of the 
embodiment of the invention just described. 

|0064) An RSM may include a ground-to-imaging function such as the polynomial described 
above with respect to prior art abstract sensor models. However, the function is not limited to 
polynomials or even to rational functions (i.e., ratios of polynomials), and may include other 
functions such as an interpolated grid of ground point-to-image point correspondences. Such 
a grid can be formed by evaluating the rigorous sensor model at a grid of ground locations 
within the image ground footprint at different elevation planes. The resultant ground point- 
image point correspondence grid can then be interpolated at the exact ground point location 
desired, resulting in the corresponding image coordinates. Quadratic interpolation in each of 
the three ground coordinate components is recommended. For the purposes of clarity and 
example, the remainder of this disclosure will refer to the case where the function is a 
polynomial; however, all of the concepts presented are equally applicable to the case where 
the function is the interpolated grid just described. 

[00651 Coefficients of the polynomial are typically generated as in the prior art; i.e., by 
fitting the coefficients to the support data of a particular image sensor. This may be done by 
fitting the parameters to a ground point-image point correspondence grid that corresponds to 
an image footprint and is mathematically generated (predicted) using the rigorous sensor 
model. 

[0066] As alluded to above, it is possible to select adjustable parameters of the polynomial 
that reflect support data errors in the rigorous sensor model. It should be noted that the 
adjustable parameters are selected from a plurality of possible parameters, and, in asmuch as 
they represent initially unknown support data errors, the values of the parameters will initially 
be assumed to be zero. Examples of the adjustable parameters are discussed in more detail 
below, but include, for example, a bias correction to u, a coefficient for the change in u with 
respect to x, a coefficient for the change in u with respect to y, a bias correction to v, a 
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coefficient for the change in v with respect to x, and a coefficient for the change in v with 
respect to y. Together these selected adjustable parameters can be thought of as forming an 
adjustment vector A. The vector A permits the definition and calculation of an error 
covariance associated with the RSM and hereinafter referred to as C A . Although the values 
of A are initially zero since actual values of support data errors are unknown, they specify 
parameters for a calculation of a partial derivative B A as will be discussed below. 
[0067] As will be seen, the calculation of, for example, Ca and Ba allow for a direct 
substitution for their counterparts in a rigorous sensor model calculation, and then 
conventional prior art techniques such as the M-image Extraction Algorithm can be practiced 
with the same ease and efficacy as in the prior art, but with all the additional advantages of an 
abstract sensor model. 

[0068] A replacement sensor model according to the present invention, then, can be utilized 
in the following exemplary manner, as demonstrated in Figs. 6-7. Specifically, Fig. 6 
demonstrates an exemplary implementation of one embodiment of the present invention. Fig. 
7 is a flowchart describing actions of an operator or system in the implementation of Fig. 6. 
[0069] In Fig. 6, an operator 605 of an image sensor (not shown) included in an orbiting 
satellite 610 can receive information indicated by dashed line 615. Information 615 includes 
an image of a particular image footprint and support data of the image sensor at the time the 
image was obtained, to be stored in storage 645, by way of antenna 620 and receiver 625. 
The image footprint and support data may be, for example, an image of a city or a portion of 
a city. 

[0070] Such an operator will typically have a conventional rigorous sensor model 1 1 0. 
Sensor model 1 10 is used to generate an image point-ground point correspondence grid, 
which in turn is used to determine the coefficients of the polynomial of the RSM, essentially 
as in the prior art. That is, a representative grid of ground points are chosen and input into 
the sensor model 1 10 to obtain a grid of image measurements. Note that this grid of ground 
points should be relatively extensive and representative of the image footprint, so as to 
minimize a fit error when generating the coefficients of the polynomial. 
[0071] Next, a ground point grid varying in both horizontal and vertical position within the 
image footprint, is generated, using enough ground points to support calculation of partial 
derivatives Bs. Bs represents an incremental change in the sensor model function output (u 
and v) with respect to a change in S, evaluated at each ground point X in the grid. 
[0072] Once the operator receives the image and support data, a subset of adjustment 
parameters making up adjustment vector A are selected from a plurality of pre-determined 
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,sible parameters to be used with the particular image sensor within satellite 610. Choice 
jnd application of A will be discussed below with respect to Figs. 8 A and 8B. The plurality 
of possible parameters may be thought of as being stored for selection within storage 645. 
[0073] Next, the same grid used in calculating Bs can then be used in the calculation of B A , 
the incremental change of the polynomial output (u and v) with respect to A, evaluated at 
each ground point X in the grid. Also, although the vector A represents parameters that can 
be adjusted to reflect the presence of support data errors, these errors are initially unknown, 
and so the actual initial value of the vector A will be zero. 

[0074] Once A, B A and B s are known (along with Cs), the RSM error covariance C A can be 
calculated as described in mathematical detail below. If C s corresponds to multiple images, 
each image's A, B A and B s must be computed. 

[0075] In short, RSM data file generator 655 generates the coefficients for the RSM 
polynomial, the adjustment vector A and the associated error covariance C A . These 
parameters are included in an RSM data file and associated with their corresponding image. 
[0076] Once the above calculations have been performed on the operator side, a user 635 
may wish to perform a ground point extraction from the image; for example, a user may 
request a location of a particular building or other structure within a city including in the 
image footprint. If so, the operator 605 may output the image and associated RSM data file 
to the user through transmitter 640, to be received at user's receiver 630. At this point, the 
user 635 can use the RSM data file's contents to exploit the image just as if that user had 
access to the actual rigorous sensor model. Specifically, for example, the user can make use 
of RSM processor 660 in conjunction with the RSM data file to perform error propagation 
analysis, an M-image Extraction Algorithm using properly weighted images (here assuming 
the user receives a plurality of images and associated RSM data files from operator 605), or 
even a type of triangulation in which actual values of A are solved for and the polynomial is 
actually adjusted accordingly. 

[0077] The user does not need, and will not have access to, any of the actual, physical 
support data (or its error covariance) associated with the particular image sensor, which is 
cumbersome to manipulate and may be proprietary to the operator. Moreover, RSM 
processor 660 will provide a common interface to user 635 even if that user is obtaining 
images from a plurality of operators/image sensors. Thus, the user's computational 
requirements are severely reduced, and an ease of use of the image is greatly increased. 
[0078] It should be noted that other embodiments of the invention are contemplated that 
operate in a similar manner to that illustrated in Fig. 6. For example, the generation of the 
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SM data file could take place entirely within satellite 610, and/or the image could be 
forwarded to user 635 directly from the satellite 610 as well. 

[0079] Fig. 7 illustrates the steps performed and calculations made by operator 605. 
Specifically, in step 705, the operator obtains image data and support data S. In step 710, a 
relatively large number of ground points can be used in the rigorous sensor model function to 
generate a dense ground-point image point data grid. In step 715, polynomial coefficients for 
the RSM polynomial are generated using the dense data grid previously computed, and 
rigorous sensor model error covariance Cs is obtained as part of the complete rigorous sensor 
model. 

[0080] In step 720, the ground point grid varying in both horizontal and vertical sition within 
the image footprint, referred to above, is generated to support the future calculation of partial 
derivatives Bs and B A . Thus, in step 725, partial derivative Bs can be calculated. 
[0081] Once these values are known, the parameters of adjustment vector A (having their 
initial values set at zero) can be selected from the predetermined list of possible parameters in 
step 730. Thereafter, the partial derivative B A can be calculated in step 735 using the same 
grid used in calculating Bs. 

[0082] Subsequently, the RSM error covariance Ca can be determined from Ba, Cs, and Bs in 
step 740. Once C A is known, RSM data file including the polynomial coefficients, A and C A 
can be completed and stored in step 745. Finally, when a user wishes to exploit the original 
image (or a plurality of images) the image(s) and associated RSM data file(s) can be output to 
the user in step 750. 

[0083] The above description provides an overview of the nature, function and operation of 
the present invention, particularly at the operator side. A more detailed and mathematically 
complete explanation of an RSM according to the present invention (including a 
representative technique by which an operator might generate an error covariance C A ) and 
used in an M-image Extraction Algorithm and/or a Triangulation (as might be practiced by a 
user 635 in Fig. 6) follows. 

[0084] The RSM error covariance C A can be defined similarly to C s in equation (2); i.e., 
E{sAsA T } = C A . C A includes the effects of all intra-image and inter-image sensor parameter 
uncertainties and their correlations (When multiple images are involved, the adjustment 
vector A is redefined to include the individual adjustment vectors A corresponding to each 
image. Moreover, regarding C A , only the upper triangular portion is required for each group 
of correlated images.). 
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45] The M-image Extraction Algorithm discussed at various points above is known to 
perform a simultaneous, minimum variance solution of the three dimensional locations of k 
ground points (more precisely, the best linear unbiased estimate). It is a weighted least 
squares solution, and, as already referred to, relies on an iterative linearization about an 
initial estimate of the ground locations. Corrections to the a priori sensor support data are 
not solved for in the following example; i.e., no triangulation is performed. However, it is 
assumed that the a priori sensor support data is accurate enough to support the partial 
derivative calculations, a reasonable assumption for most sensors. The algorithm as 
expressed in equation (5) utilizes m (greater than or equal to 1) images that contain the 
multiple (k) ground points. It properly accounts for all error sources and their correlations. 

X = X 0 + 8X , where SX = PB T X WZ , P=[pf + B T X WB X , 

W=[z M +B s C s BlY , and Z=M-M 0 ; (5) 

iterate as necessary, i.e., set X 0 to current solution X and redo solution. 

[0086] The following definitions are applicable to the above equations: 

(1) X 0 (3k x 1) is the a priori estimate of the ground point locations and Po (3k x 3k) its a 
priori error covariance (typically a large valued diagonal matrix). 

(2) 8X (3k x 1) is the unbiased best estimate of the correction to X 0 . X = X 0 + SX is the 
unbiased best estimate of the ground point location and P (3k x 3k) its a posteriori error 
covariance. 

(3) M is the image measurements corresponding to k ground points in m images (2mk x 1) 
and Mo the original sensor model's predicted measurements, a function of Xo and the sensor 
parameters S (mn x 1) associated with m images. 

(4) S M (2mk x 2mk) is the image mensuration error covariance (typically a diagonal matrix 
with (.5 pixels) 2 down the diagonals); note that mensuration errors reflect imperfect 
identification/measurement of pixel coordinates (u,v) and are the minor term in W. C s (mn x 
mn) is the a priori error covariance of the sensor support data, as already defined. 

(5) Bx (2mk x 3k) is the partial derivative matrix of M with respect to X. B s (2mk x mn) is 
the partial derivative matrix of M with respect to support data vector S. Both B x and Bs can 
be generated either analytically or numerically from the original sensor model about the Mo 
operating point. 
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[0087] The same weighted least squares algorithm of equation (5) is also applicable when 
using the replacement sensor model by simply making the following substitutions as shown 
in equation (6): 

C s ^ C A (mqxmq),B s -» B A (2mk x mq), and M 0 -> M* (2mk x 1) (6) 



[0088] B A is the partial derivative matrix of M with respect to A, where the dimension of A 
is (qm x 1), i.e., q adjustable parameters per image. B A as well as B x can be computed either 
analytically or numerically from the replacement sensor model's polynomial about the M<f 
operating point. M* is the replacement sensor model's predicted measurements, computed 
from the replacement sensor model's polynomial as a function of Xo- 
[0089] The solution a posteriori error co variance of equation (7) quantifies accuracy 



=E\sXs: 



Pn - Pu 
Pkx - P* 



(7) 



[0090] Equation (8) represents the absolute accuracy of ground point i. 



P xi =E{sX,eXf 



a„ a„ a z , 



2 ~ 2 

xy °xz 
ff lr ff lv °\z 



(8) 



[0091] Finally, equation (9) represents the relative accuracy between ground points i and j. 



P; ^P+P^P.-PjrE^sK, -eX ] )(sX i -eX,/} 



(9) 



[0092] In short, equations (5)-(9) demonstrate the use of one embodiment of the invention in 
the M-image Extraction Algorithm. Thus, once the user has measured the image coordinates 
of the ground object(s) (as is conventional), he or she needs only the RSM processor 660 of 
Fig. 6 and the RSM data file to use the invention in this manner, and need only make the 
substitutions of equation (6) to do so. All calculations after that substitution are computed as 
in the prior art. 
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[0093] The present invention can also be used to perform a process akin to triangulation, as 
discussed below and demonstrated with respect to Figs. 8 A and 8B. The selection and use of 
the parameters making up adjustment vector A are also discussed in this context, although of 
course the vector A can be used in the present invention even when its value(s) are not solved 
for and remain zero, as demonstrated above. 

[0094] A covariance-based weighted least squares adjustment with the adjustment vector's a 
priori error covariance providing the proper weight for the adjustment vector's current value 
can be made to solve for non-zero value(s) of A that reflect corrections for errors in the 
original support data. In this implementation, the replacement sensor model's refinement 
should be virtually mathematically equivalent to a corresponding triangulation, since the 
replacement solution (assuming the appropriate adjustable polynomial parameters) replicates 
the original solution for arbitrary sets of ground points. Once such non-zero values for A are 
determined, A can be applied to the RSM polynomial as shown in Figs. 8A and 8B. 
Similarly, if one who obtains the image performs such a calculation, then the results can be 
forwarded on to others wishing to perform ground point extractions, who will now be able to 
do so more accurately. 

[0095] Specifically, in Fig. 8 A, ground position vector X is input into polynomial 810, where 
polynomial 810 has a form and coefficients determined in the manner described above with 
respect to conventional replacement sensor models. Thus, polynomial 810 outputs first 
image coordinates (u',v'), which are then modified by adjustment vector 820. In this way, 
the corresponding image coordinate adjustments (u", v") are added to (u',v') to yield the 
image coordinate outputs (u,v). This type of adjustment is referred to as an image space 
adjustment, since it occurs on the image side of the polynomial. Also, it should be noted that 
in this scenario it is mathematically equivalent to simply change the actual coefficients of the 
polynomial 810 and perform a single operation. 

[0096] The polynomial with an image space adjustment is of the form shown in equation 
(10): 

u = (aO + alx + a2y + a3z + a4xy + ..) + (AaO + Aalx + Aa2y) 

v = (bO + blx + b2y + b3z + b4xy + ..) + (AbO + Ablx + Ab2y) , and 

t r i ( 10 > 

A T s [AaO Aal Aa2 AbO Abl Ab2j, where 

AaO, .. , Ab2 are the additional adjustment parameters. 
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10097] As indicated above, equation (10) defines an adjustment vector A to include a set of 
six adjustment parameters. Although the low-order corrections effected by these adjustment 
parameters were determined to be well-suited for rigorous error propagation, any set of low 
order corrections can be defined. (Note that low order corrections can also be similarly 
applied to a rational polynomial, i.e., the ratio of two polynomials. However, corrections are 
only applied to the numerator.) 

[0098] In contrast, Fig. 8B demonstrates an adjustment vector 830 being used to adjust the 
ground position vector X. Thereafter, adjusted vector X' is input into polynomial 810. This 
type of adjustment is referred to as a ground space adjustment, since it occurs on the ground 
side of the polynomial. 

[0099] The polynomial with a ground space adjustment is of the form shown in equation 
(11): 

u = (a0 + alx'+a2y , +a3z , +a4x'y'-h.) 



v = (b0+blx , +b2y , +b3z'+b4x'y'+..) , where 
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(11) 



A T = [Ax Ay Az a (3 y] 



[0100] Further note that the definition of the adjustment vector A, either via equation (10) or 
equation (11), allows for the computation of the partial derivative matrix B A used in equation 
6. 

[0101] A six element affine transformation consisting of small ground space corrections to 
the polynomial's independent variables was assumed in this example. These corrections are 
defined as the elements of the adjustment vector A. Again, although the above 6 corrections 
were determined to be well-suited for error propagation, other ground space adjustment 
parameters can be defined as would be apparent. 

[0102] Note also that the coordinate system of which the adjustments are a function can be 
more generally represented as a local Cartesian system, centered at the image footprint, with 
z-axis aligned in the image locus direction, x-axis aligned in the image scan direction, and y- 
axis completing the right-hand coordinate system. 
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^03] As referred to above, the parameters of vector A used above can be chosen from a 
larger list of possible parameters, for operation on the image side or ground side of the 
polynomial. On the image side, for example, at least the following twelve possibilities exist: 
a (constant) bias correction to u, a coefficient for a change in u with respect to x, a coefficient 
for a change in u with respect to y, a coefficient for a change in u with respect to x 2 , a 
coefficient for a change in u with respect to xy , and a coefficient for a change in u with 
respect to y 2 . Similar six corrections exist for v on the image side. 
[0104] On the ground side, at least the following seven corrections exist from which 
parameters of A may be chosen: a (constant) bias correction to x, y or z, an Euler angle 
rotation about those three axes (represented by a,/3,y in equation (11)), and a scale factor 
adjustment to X when translating coordinate values x, y and z as described with respect to 
equation (11). For some images, six parameters from the above list excluding the z 
correction work well. 

[0105] Having now demonstrated exemplary techniques for the use of this embodiment of the 
present invention, the following is an exemplary derivation of the error covariance C A from 
rigorous sensor model error covariance Cs and associated partial derivatives. 
[0106] In general, as referred to at the beginning of the Detailed Description, the error in the 
polynomial adjustment vector ( s A) is to closely approximate the effects of the error in the 
original sensor support data state vector {s S) on an arbitrary m-image ground point solution. 
That is, the image measurement error due to s A is to closely approximate the image 
measurement error due to e S, as shown in equation (12): 

B A eA=B s eS (12) 

[0107] Equivalently, since the sensor model(s) are mean zero statistical models, equation 
(13) can be written as: 

E{(B A sA)(B A eAf } = E{(B s sS)(B s sS) T }, or B A C A B T A = B S C S B T S . (13) 

Thus, equation (13) implies that the ground point solution using the replacement sensor 
model closely approximates the ground point solution using the original sensor model, as 
shown below in equation (14). Note that equation (14) corresponds to the equations of the 
M-image Extraction Algorithm, as documented earlier in equation (5). 
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replacement = [P~ l +B T X (Z U + B A C A B T A )- l B x J- l B T x (Z M + B A C A B T A )-'(M - M* ) 
i /P" 1 + B X (E M + B s C s B T s )- l B x r'B T x (Z M + B s C s B T s )~ l (M - MJ = SX _original , (14) 
where M„ =M 0 due to negligible polynomial fit error. 

[01081 Thus, C A must be generated such that equation (13) is satisfied. Since B A can not be 
inverted in general, and because equation (13) is to hold at arbitrary ground locations X, the 
generation of C A is accomplished as demonstrated in equations (15-17) 

C A = (B*f )(B* s C s B*s )(By f , where the pseudo - inverse of B A equals 
B J = (B A T B A )~ l B A T . Thus, we also have 

C A = (B*J B* A ) ] B* A T (B* S C S B*$ )B* A (B*J B* A )~ T . (15-17) 

[0109] The superscript * on the partial derivative matrices B A and B s indicate that the partial 
derivatives correspond to a grid of ground point locations, varying both horizontally and 
vertically, within the image footprints. The footprints correspond to all (correlated) images 
whose support data errors are represented by Cs. Enough ground points are chosen to 
capture the variation of the partial derivatives over the footprints, as well as to insure that B * A 
is full rank. In addition, the number of replacement sensor model adjustable parameters per 
image is typically less than or equal to the number of sensor parameters per image. This 
facilitates B^ being full rank and is a necessary condition for a positive definite C A 
(discussed later in more detail). Also, it is assumed that the a priori sensor support data 
errors are not so large as to render the partial derivative computations invalid. 
[0110] In general, equations (15-17) correspond to multiple (m) images. Assuming n sensor 
parameters per image i (i=l,..,m) and contained in Si, q RSM adjustment parameters per 
image i and contained in Ai, p ground points in image i's ground point grid, C s is a mn x mn 
multi-image sensor parameter error covariance matrix, and C A is a mq x mq multi-image 
RSM adjustable parameter error covariance matrix. In addition, B* s , is defined as the 2p x n 
matrix of partial derivatives of image i measurements with respect to Si applicable over the 
image i ground point grid, and B* Al as the 2p x q matrix of partial derivatives of image i 
measurements with respect to Ai applicable over the image i ground point grid. Thus, 
components of equation (17) can be further detailed as follows: 
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\B A ]B AX y x B] 



0 



, a block diagonal matrix; 



0 




0 




, a block diagonal matrix. 



(18-19) 



0 



[Olll] The solution of equations (15-17) can also be generalized as follows. The problem of 
solving for C A such that the matrix norm difference of equation (20) is minimized: 



The particular norm is the Frobenius norm. The Frobenius norm of a matrix is the square 
root of the sum of its elements squared. The solution to this minimization problem is 
demonstrated by equation (21): 



[0112] In equation (21), the superscript + indicates the Moore-Penrose Generalized Inverse. 
The Moore-Pensrose Generalized Inverse of a matrix is based on the Singular Value 
Decomposition of the matrix. 

[0113] When B^is full rank, becomes the pseudo-inverse B^ p , and equation (21) is 
equivalent to equations (15-17). Thus, the solution for C A of equation (21) is only required 
for the pathological case when B^ is not full rank. Also, when B^ is full rank, the solution 
of equations (15-17) minimizes the Frobenius norm as well. 

In short, for a given adjustment vector A (definition), Ca is computed such that its 
projection to image space closely approximates the projection of the support data error 
covariance of the rigorous sensor model onto image space, as was discussed at the beginning 
of the Detailed Description. In addition, this process can be used to select an optimal A 
(definition). In particular, an A can be selected such that its corresponding minimum 
Froebenius norm value is also minimal compared to the rmnimum Froebenius norm values 



\b' a c a bj -b;c s b; t \\ f 



(20) 



Ca - B^(B s C s Bj )B A T , 



(21) 
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computed for other A candidates. The candidates can be selected manually, or can be 
selected automatically using an algorithm which selects combinations of A components from 
the list of all possible candidates. In addition, the above selection process may be augmented 
with a positive definite constraint on the A and corresponding C A candidates. That is, C A 
must also be a positive definite matrix (discussed below in more detail). 
[0114] Finally, an alternate solution is presented that ensures C A is positive definite. That is, 
the solution of equations (15-17) ensures a positive semi-definite matrix, which is required 
for the M-image Extraction Algorithm. For a reasonable selection of adjustable parameters, 
and with a number of elements no greater than the corresponding numbers of sensor support 
data error parameters, C A is typically positive definite as well. However, a triangulation 
solution requires that C A be positive definite (and therefore invertible as well). Equation (22) 
ensures C A is positive definite, but is typically only used when the previous algorithm does 
not yield a positive definite matrix. 

[0115] Equation (22) requires that the number of ground points in the evaluation grid must 
be limited such that B \ is approximately a square matrix. If not, C A can become too small 
when the rank of B \ exceeds the rank of B * A . 

C A = [C- A l + Bj(C z + BlC s Bj)-'B\r (22) 

[0116] In equation (22), C z is a diagonal matrix with a small amount of measurement 
uncertainty ((0.1 pixels) 2 ) down the diagonals. C A0 is a diagonal matrix representing a large 
amount of uncertainty in A. For example, C A o may represent an amount of uncertainty 
equivalent to (2000m) 2 ground uncertainty for each element of A. 

[0117] In conclusion, the replacement sensor model of the present invention may be used in 
place of the original, rigorous sensor model. This replacement sensor model provides for 
adjustability of the ground-to-image function and significantly improves error propagation 
information, such that the error propagation information is virtually equivalent to that 
available in a complete original sensor model. 

[0118] This allows for optimal image exploitation when using the new replacement sensor 
model. In addition, the new replacement sensor model retains the overall user community 
benefits afforded by replacement sensor models in general. These include low development 
and maintenance costs, a common interface, and the potential for wider dissemination of 
proprietary imagery. 
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[0119] While this invention has been described in various explanatory embodiments, other 
embodiments and variations can be effected by a person of ordinary skill in the art without 
departing from the scope of the invention. 
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